Deep learning models often have more parameters than observations, and still perform well. This is sometimes described as a paradox. In this work, we show experimentally that despite their huge number of parameters, deep neural networks can compress the data losslessly even when taking the cost of encoding the parameters into account. Such a compression viewpoint originally motivated the use of variational methods in neural networks. However, we show that these variational methods provide surprisingly poor compression bounds, despite being explicitly built to minimize such bounds. This might explain the relatively poor practical performance of variational methods in deep learning. Better encoding methods, imported from the Minimum Description Length (MDL) toolbox, yield much better compression values on deep networks.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Knowing how to efficiently balance between exploring unfamiliar parts of an environment and exploiting currently available knowledge is an essential ingredient of anyintelligent organism.

Skill learning offers one way of identifying these regularities by decomposing pre-collected experiences intoasequence ofskills.

A single capsule is wired to represent a single entity, and its vector/matrix may learn to encode its pose w.r.t the coordinate frame imposed byanobserver.